Repository navigation
fix(batch_evaluation): route batch_evaluation through v2 observations API - #1898
passionworkeer wants to merge 7 commits into
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Resolve the metadata omission and cross-page trace collapsing issue; correct the inaccurate field-group comment.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (2)
What changed in this PR
Migrates batch_evaluation to the v2 observations API for Langfuse events_only deployments.
Changes:
- Adds cursor-based v2 observation fetching and field mapping.
- Collapses observations into trace representatives.
- Adds unit coverage for fetching, grouping, pagination, and field selection.
| File | Summary | Findings |
|---|---|---|
tests/unit/test_batch_evaluation_fetch.py |
Tests v2 fetching, grouping, filtering, pagination, and field mapping. | No findings. |
langfuse/batch_evaluation.py |
Implements v2 observation fetching and trace collapsing. | Critical: default fields omit metadata. Moderate: trace collapsing is page-local and can duplicate traces. Nit: comment incorrectly describes supported field groups. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # The legacy ``/api/public/traces`` ``io`` / ``scores`` / ``observations`` / | ||
| # ``metrics`` field groups are not selectable on the v2 read endpoint because | ||
| # it returns one observation at a time. | ||
| _DEFAULT_V2_OBSERVATION_FIELDS = "core,basic,io,usage,model,trace_context" |
There was a problem hiding this comment.
Fixed in 6021c60 — metadata added to the default v2 field groups.
| def _collapse_observations_to_traces( | ||
| observations: List[ObservationV2], | ||
| ) -> List[ObservationV2]: | ||
| """Collapse a flat list of observations to one observation per trace_id. | ||
|
|
||
| The v2 observations endpoint has no trace-level read; this helper takes | ||
| whatever observations the page returned and returns one representative | ||
| per trace, preferring the observation that the server already marked as | ||
| the root (``is_root_observation=True``) and falling back to the first | ||
| observation seen for that trace. |
There was a problem hiding this comment.
Fixed in 6021c60 — the runner now tracks seen trace IDs across cursor pages.
… API BatchEvaluationRunner._fetch_batch_with_retry used the v3 read APIs (GET /api/public/traces and GET /api/public/observations via legacy.observations_v1) to fetch items for batch evaluation. Both endpoints return HTTP 400 on Langfuse platform v4 events_only deployments, so every batch evaluation against an event-store-backed self-hosted Langfuse v4 fails on the first page (langfuse/langfuse#1861). This change routes the runner through GET /api/public/v2/observations, which is the only read path that works on v4 events_only (and remains available on v3, with the v3 endpoint scheduled for removal on 2026-11-16). The pagination state switches from page-based to cursor-based. For scope=traces, observations are collapsed to one representative per trace (preferring is_root_observation=True), since the v2 endpoint has no trace-level read. The MapperFunction protocol accepts the v2 ObservationV2 in addition to the legacy TraceWithFullDetails / ObservationsView, so existing mappers keep working with the input/output/metadata fields they already read.
…e order
The v2 observations endpoint may not return observations with the trace root
first. The previous 2-dict implementation skipped the trace after its first
appearance, so a non-root observation could prevent the root from being
picked even when the root appeared later in the page.
Switch _collapse_observations_to_traces to a single-dict form that replaces
the chosen representative whenever a strictly better candidate appears.
Also expand unit tests:
- test_fetch_batch_prefers_root_observation_regardless_of_page_order
exercises the bug above.
- four test_v2_observations_fields_* tests cover the
fetch_trace_fields to v2 field-groups translation.
Cursor-loop comment is corrected to match the actual behaviour (the
implementation has always stopped on an empty page; the previous comment
about preserving the cursor on transient empty pages was wrong).
571f327 to
26eceed
Compare
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if scope == "traces": | ||
| items = cast( | ||
| List[Union[TraceWithFullDetails, ObservationsView, ObservationV2]], | ||
| _collapse_observations_to_traces(list(response.data)), # type: ignore[arg-type] | ||
| ) |
There was a problem hiding this comment.
scope="traces", this path returns an ObservationV2, but the runner still uses item.id as the trace ID. That is the observation’s ID, not item.trace_id, so trace-level scores and optional tags are sent with the wrong trace ID and may be missing from the intended trace.
Knowledge Base Used: Experiments and batch evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/batch_evaluation.py
Line: 1276-1280
Comment:
**Observation IDs used as trace IDs** When `scope="traces"`, this path returns an `ObservationV2`, but the runner still uses `item.id` as the trace ID. That is the observation’s ID, not `item.trace_id`, so trace-level scores and optional tags are sent with the wrong trace ID and may be missing from the intended trace.
**Knowledge Base Used:** [Experiments and batch evaluation](https://app.greptile.com/personal-org-4986/-/custom-context/knowledge-base/langfuse/langfuse-python/-/docs/experiments-and-batch-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 6021c60 — _get_item_id returns trace_id for scope='traces'.
| # The legacy ``/api/public/traces`` ``io`` / ``scores`` / ``observations`` / | ||
| # ``metrics`` field groups are not selectable on the v2 read endpoint because | ||
| # it returns one observation at a time. | ||
| _DEFAULT_V2_OBSERVATION_FIELDS = "core,basic,io,usage,model,trace_context" |
There was a problem hiding this comment.
metadata field group; io supplies only input and output. With these defaults, observation metadata is absent, so a mapper that reads it receives None. The documented mapper’s observation.metadata.get(...) path then fails for non-generation observations.
| _DEFAULT_V2_OBSERVATION_FIELDS = "core,basic,io,usage,model,trace_context" | |
| _DEFAULT_V2_OBSERVATION_FIELDS = "core,basic,io,metadata,usage,model,trace_context" |
Knowledge Base Used: Experiments and batch evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/batch_evaluation.py
Line: 41
Comment:
**Default fields omit metadata** The v2 API requires a separate `metadata` field group; `io` supplies only input and output. With these defaults, observation metadata is absent, so a mapper that reads it receives `None`. The documented mapper’s `observation.metadata.get(...)` path then fails for non-generation observations.
```suggestion
_DEFAULT_V2_OBSERVATION_FIELDS = "core,basic,io,metadata,usage,model,trace_context"
```
**Knowledge Base Used:** [Experiments and batch evaluation](https://app.greptile.com/personal-org-4986/-/custom-context/knowledge-base/langfuse/langfuse-python/-/docs/experiments-and-batch-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 6021c60 — metadata added to the default v2 field groups.
| if scope == "traces": | ||
| items = cast( | ||
| List[Union[TraceWithFullDetails, ObservationsView, ObservationV2]], | ||
| _collapse_observations_to_traces(list(response.data)), # type: ignore[arg-type] | ||
| ) |
There was a problem hiding this comment.
Knowledge Base Used: Experiments and batch evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/batch_evaluation.py
Line: 1276-1280
Comment:
**Traces repeat across pages** If a trace’s observations span cursor pages, this helper chooses a representative independently on each page. The runner evaluates the trace again on the next page; a child on one page and the root on another can therefore produce separate evaluations for the same trace instead of one.
**Knowledge Base Used:** [Experiments and batch evaluation](https://app.greptile.com/personal-org-4986/-/custom-context/knowledge-base/langfuse/langfuse-python/-/docs/experiments-and-batch-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 6021c60 — the runner now tracks seen trace IDs across cursor pages.
| if scope == "traces": | ||
| items = cast( | ||
| List[Union[TraceWithFullDetails, ObservationsView, ObservationV2]], | ||
| _collapse_observations_to_traces(list(response.data)), # type: ignore[arg-type] | ||
| ) |
There was a problem hiding this comment.
scope="traces", the selected ObservationV2 has start_time but no timestamp. The runner consequently saves an empty resume timestamp, then applies a timestamp > "" filter to the v2 observations request. The resume token cannot reliably continue from the last processed item.
Knowledge Base Used: Experiments and batch evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/batch_evaluation.py
Line: 1276-1280
Comment:
**Trace resumes lack a timestamp** If a later page fails during `scope="traces"`, the selected `ObservationV2` has `start_time` but no `timestamp`. The runner consequently saves an empty resume timestamp, then applies a `timestamp > ""` filter to the v2 observations request. The resume token cannot reliably continue from the last processed item.
**Knowledge Base Used:** [Experiments and batch evaluation](https://app.greptile.com/personal-org-4986/-/custom-context/knowledge-base/langfuse/langfuse-python/-/docs/experiments-and-batch-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 6021c60 — uses the observation's start_time and the startTime filter column.
| response = self.client.api.observations.get_many( # type: ignore[union-attr] | ||
| cursor=cursor, | ||
| limit=limit, | ||
| filter=filter, | ||
| request_options={"max_retries": max_retries}, | ||
| fields=v2_fields, | ||
| ) |
There was a problem hiding this comment.
scope="traces", the original filter now goes to the observations endpoint unchanged. A name = "checkout" filter that previously selected traces by name now selects observations by name; a matching trace whose root observation has a different name is omitted, while an observation in an unrelated trace can match.
Knowledge Base Used: Experiments and batch evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/batch_evaluation.py
Line: 1264-1270
Comment:
**Trace filters change meaning** For `scope="traces"`, the original filter now goes to the observations endpoint unchanged. A `name = "checkout"` filter that previously selected traces by name now selects observations by name; a matching trace whose root observation has a different name is omitted, while an observation in an unrelated trace can match.
**Knowledge Base Used:** [Experiments and batch evaluation](https://app.greptile.com/personal-org-4986/-/custom-context/knowledge-base/langfuse/langfuse-python/-/docs/experiments-and-batch-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| if max_items is not None and total_items_fetched >= max_items: | ||
| has_more = True # More items exist but we're stopping | ||
| break |
There was a problem hiding this comment.
max_items is reached on a page whose v2 cursor is None, this branch sets has_more back to True. The result then reports has_more_items=True even though the server says there are no further results, which can prompt callers to look for work that does not exist.
Knowledge Base Used: Experiments and batch evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/batch_evaluation.py
Line: 1169-1171
Comment:
**Final page reports more items** When `max_items` is reached on a page whose v2 cursor is `None`, this branch sets `has_more` back to `True`. The result then reports `has_more_items=True` even though the server says there are no further results, which can prompt callers to look for work that does not exist.
**Knowledge Base Used:** [Experiments and batch evaluation](https://app.greptile.com/personal-org-4986/-/custom-context/knowledge-base/langfuse/langfuse-python/-/docs/experiments-and-batch-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 6021c60 — has_more stays false when the server already returned cursor=None.
Copilot and Greptile flagged six concerns against the previous two commits; this commit resolves them in one place. - Default v2 field groups: add ``metadata``. The ``io`` group only carries input/output strings; ``metadata`` is a separate v2 field group and was missing from the defaults, so every ``ObservationV2`` passed to a mapper had ``metadata=None``. The defaults are now ``core,basic,io,metadata,model,usage,trace_context``. - Cross-page trace collapse state: ``BatchEvaluationRunner`` now tracks the trace IDs already collapsed on an earlier page in ``self._seen_trace_ids`` and ``_collapse_observations_to_traces`` accepts an optional set so the same trace is never evaluated twice across cursor pages. - ``_get_item_id`` for ``scope=traces``: the item is now an ``ObservationV2`` whose ``id`` is the observation ID, not the trace ID. Use ``trace_id`` instead so downstream score-create calls attach to the intended trace. - ``_get_item_timestamp`` and ``_get_timestamp_field_for_scope``: use the observation ``start_time`` as a proxy for the trace timestamp; the v2 filter column is ``startTime``. Resume tokens continue to work; legacy ``TraceWithFullDetails.timestamp`` is still consulted as a fallback. - Final-page ``has_more`` report: do not set ``has_more = True`` when the server already returned ``cursor=None``; the previous code reported ``has_more_items=True`` even though the server said there were no further results. - ``_translate_trace_filter``: rewrite ``name`` -> ``traceName`` and ``timestamp`` -> ``startTime`` in v3-shaped trace filters so a trace name filter selects the same set of traces on the v2 endpoint as it did on v3. The translation only runs for ``scope=traces``. Tests: 6 new unit tests cover the cross-page collapse, filter translation, the missing-metadata field group, the scope-aware ``_get_item_id`` and ``_get_item_timestamp`` paths, and the post-cursor-empty max-items path. All 16 tests in tests/unit/test_batch_evaluation_fetch.py pass; the larger unit suite shows 695 passed, 2 skipped, and the same 18 pre-existing test_prompt.py credential errors that exist on base_sha.
|
cc @hassiebp for review when you have a moment. Review feedback is addressed in 6021c60 and 6865554 (filter translation, trace-id usage, cross-page dedupe, resume timestamps, field defaults, has_more edge cases — details in the commit messages). One open question: would you prefer the v2 read path as the default (v3 reads are deprecated with EOL 2026-11-16) or gated behind a flag? Happy to split a flag-gated rollout out as a follow-up if that's easier to land. |
… paths Follow-up to the review-fix commit, from a local pre-merge adversarial review (12-item checklist run by a reviewer subagent). Docstring corrections (no runtime change): the public docstrings for run_batched_evaluation and BatchEvaluationRunner.run_async still claimed the runner reads the v3 endpoints and "is not yet supported with platform v4" — the exact opposite of what this PR does. They now describe the v2 observations endpoint, cursor pagination, the per-trace collapse, and the v2 field-group semantics of fetch_trace_fields (legacy-only groups are dropped; user-supplied groups are merged into the defaults). Hardening: - Trace filter translation now also rewrites ``id`` to ``traceId``; a v3 trace-id filter previously matched against v2 observation ids and silently returned an empty selection. - An empty page no longer leaves ``has_more`` set when the server also returned a cursor; the run is treated as exhausted, matching the pre-migration behaviour. - The fetch-failure early return now flushes the client before building the resume token, so scores already created for earlier pages are not lost with the process (pre-existing gap on base, one-line fix). Tests: five new run_async-level tests cover the max-items-on-last-page reporting (completed=True / has_more_items=False), max-items mid-stream (has_more_items=True), the empty-first-page completion, the _seen_trace_ids reset between consecutive runs, and the flush before the resume-token return. The filter-translation test now asserts the ``id -> traceId`` rewrite. 21 tests in the file; full unit suite 700 passed, 2 skipped, 18 pre-existing test_prompt.py credential errors.
The v2 observations endpoint pages by cursor over observations, not traces, so one trace's root and its children can straddle a page boundary. The collapse helper chose a representative within a page, so whichever page arrived first fixed the representative for the whole run: if the child came first, the trace was evaluated on the child and `_seen_trace_ids` then suppressed the root on every later page. That made the root preference best-effort exactly when a trace spans pages. `scope='traces'` now narrows the request itself to root observations, so the choice is made once for the whole run and each page yields at most one observation per trace. The condition goes into the `filter` string rather than the `is_root_observation` query parameter because the endpoint documents that `filter` takes precedence over query-parameter filters; a caller condition on the same column is dropped rather than appended, since a filter excluding roots would otherwise combine with the narrowing to return nothing. A `scope='traces'` filter that is not a JSON array cannot be merged with the root condition, so it is now rejected in `run_async` before the fetch loop. The fetch loop's `except Exception` would otherwise swallow it into a `completed=False` result whose resume token carries an empty timestamp bound, which is a guaranteed 400 on the next run with nothing pointing at the filter. `scope='observations'` forwards the caller's filter unchanged and is unaffected. Tests: the six new cases fail against the previous implementation and pass against this one. `tests/unit/test_batch_evaluation_fetch.py` 31 passed, `tests/unit` 710 passed / 2 skipped. The 18 errors in `tests/unit/test_prompt.py` are environmental (no LANGFUSE_PUBLIC_KEY) and reproduce identically on the unmodified base commit. ruff check, ruff format and mypy clean. Not verified here: the `isRootObservation` filter against a live v4 events_only deployment, and the e2e suite.
|
Both review threads asked for root selection that is global to the run rather than per page, and that is now in The cross-page dedup I added earlier only stopped a trace being evaluated twice; it did not fix which observation was chosen. If a page carrying a child arrived before the page carrying its root, the child became the representative and the
This also improves the resume boundary noted in the description: Two costs, both stated in the description rather than left implicit:
One gap I want to be explicit about: this verifies the filter shape against the generated client's documented schema, not against a live v4 |
The unit tests pin the request the runner sends, but only a live deployment can show that the isRootObservation filter is honoured, and that the cross-page bug this replaced was real. Both are now asserted end to end in a new e2e file. Measured against a self-hosted Langfuse 4.42.0 events_only stack (postgres + clickhouse + redis + minio + web + worker), confirmed events_only first: the v3 read endpoints return 404 events_only while /api/public/v2/observations returns 200. A trace seeded with 12 children, read back at page_size 2, gave: pre-fix (page-local collapse, no root filter): picked a child observation post-fix (same helper, root-only pages): picked the server-side root Repeated across runs the server ordering varied -- the root was sometimes on page 1, sometimes not -- and the pre-fix path tracked that variation while the post-fix path always returned the root exactly once. One earlier check had the root on page 1, which is exactly the ordering that hides the bug, so the test records that ordering rather than asserting a fixed one. Also verified the filter is not a no-op in either direction: the equality condition returns only the flagged root, and the negated condition returns the children and excludes it. The file needs no shard registration; select_e2e_shard.py assigns it to shard 0. EOF git log --oneline -2
|
Closing the gap I flagged in the previous comment: The deployment was confirmed events_only before anything was concluded from it — The measurement that matters, on a trace with one root and twelve children read back at
So the defect was real, not theoretical: with the root filter removed, the trace was evaluated on a child observation. What is also visible in the logs is that the pre-fix outcome depended on the server's row ordering — the root was sometimes on page 1, and pre-fix then picked it by luck. Across repeated runs of the new e2e test the ordering varied and the pre-fix path tracked that variation, while the root-filtered path returned the root every time. That is why the test records the observed ordering rather than asserting a fixed one; a test that pinned the lucky ordering would pass against the broken code. I also checked the filter is not a no-op in either direction: New file One caveat on scope, unchanged from before: this verifies the filter and the root selection, not every other path in this PR. The v4 read path more broadly was verified against a real self-hosted deployment in an earlier revision, and the remaining limitation is unchanged — a trace with no observation the server marks as a root is not returned and therefore not evaluated. |
…ong claim
Measuring the two edge cases the root narrowing rests on showed the PR body's
justification was wrong. It said "with one root per trace, every page yields at
most one observation per trace", but two sibling spans on the same trace were
both flagged isRootObservation=true by a real server. The filter does not
provide that guarantee.
What actually holds is the collapse: it reduces a page to one observation per
trace, and seen_trace_ids keeps it to one across pages. Corrected the comment in
batch_evaluation.py to say so, and to name the two cases that break the "one
root" assumption:
- Multiple flagged roots: two siblings on one trace, both flagged. Added a test
asserting the collapse still yields one representative, and one evaluation
across pages.
- No flagged root: a span ingested over OTLP with a parent id that does not
exist never passes through this SDK's app-root marking, so nothing is flagged.
Added a test asserting the filter returns nothing and scope='traces' never
evaluates that trace -- the documented consequence, now pinned.
The observation write endpoints refuse creates on an events_only deployment
(/api/public/observations answers 405, /api/public/ingestion answers "Event type
not accepted ... only accepts score events"); the error names the OTLP path as
the events_only-compatible route, so that is what the no-root test uses.
Also corrected the run_async mapper example, which read metadata={"trace_id":
trace.id} -- for scope='traces' the item is an ObservationV2, so .id is the
observation ID while scores attach to trace_id.
Five e2e tests pass against the live events_only stack; unit suite 31 passed;
ruff and mypy clean.
|
Correcting something I got wrong in the previous two comments and in the PR description. I justified the root narrowing with "with one root per trace, every page yields at most one observation per trace". Measuring the edge cases on the live stack at So the filter does not provide the one-representative guarantee. What actually provides it is the collapse — it reduces a page to one observation per trace, and the cross-page Both cases that break the assumption are now covered:
The second row is the limitation I flagged before, now pinned by a test rather than only described. Producing it needed the OTLP endpoint: Also fixed a stale doc example:
|
|
Re-ran every remaining path of this PR end to end on the same self-hosted 4.42.0 events_only stack, at the current head
Two measurements correct things this description previously claimed:
Two recorded observations, neither changed by this PR:
Corpus and commands: 4 traces, root as a GENERATION carrying |
What does this PR do?
Fixes #1861 —
batch_evaluationfails on Langfuse v4 events_only deployments becauseBatchEvaluationRunner._fetch_batch_with_retrystill calls the v3 read endpoints (GET /api/public/tracesand legacyGET /api/public/observations), both of which return:This PR routes both
scope='traces'andscope='observations'throughGET /api/public/v2/observations, which is the only read path that works on events_only deployments and is also the direction Langfuse is taking as v3 deprecates toward the 2026-11-16 EOL.Implementation:
_fetch_batch_with_retrynow callsclient.api.observations.get_manywithcursor/limit/filter/fields, replacing both thetrace.listand thelegacy.observations_v1.get_manycalls. The page-modepageparameter is replaced with cursor pagination (response.meta.cursor)._v2_observations_fieldstranslates the legacyfetch_trace_fieldsargument into the v2 field-groups understood by/api/public/v2/observations. Caller-supplied groups are merged with the v2 defaults; legacy-only groups (observations,scores) are dropped because the v2 endpoint returns one observation at a time and does not expose trace-level sub-collections._collapse_observations_to_tracescollapses a page of v2 observations to one representative per trace, preferring the one the server already marked as the root (is_root_observation=True) regardless of its position in the page.scope='traces'requests are narrowed server-side to root observations, so the root choice is made once for the whole run rather than per cursor page. See "Root selection across cursor pages" below.MapperFunction's Protocol union now includesObservationV2additively. The existing user-supplied mapper signature stays the same; mappers that readitem.input/item.output/item.metadatawork unchanged.meta.cursor=None.Root selection across cursor pages
The v2 endpoint pages by cursor over observations, not traces, so one trace's root and its children can straddle a page boundary. Collapsing each page independently made the root preference best-effort exactly when it mattered: if the page carrying a child arrived first, the child became the representative, and the cross-page
seenset then suppressed that trace on every later page — so the root was never evaluated at all.scope='traces'therefore asks the server for roots by merging anisRootObservation = truecondition into the v2filter, which makes the representative a global choice rather than a per-page one. Two details that follow from the endpoint's documented behaviour:filterstring, not theis_root_observationquery parameter, because the endpoint documents that a suppliedfiltertakes precedence over query-parameter filters — the query parameter would be silently ignored alongside a caller filter.The filter is not by itself a guarantee of one representative per trace, and both of the cases that break that assumption are covered by tests:
isRootObservation=true. What still holds is the collapse: it reduces a page to one observation per trace, and the cross-pageseenset keeps it to one across pages, so such a trace is still evaluated exactly once.is_root_observationisOptional[bool]on the model, and traces ingested by other clients — an OTel collector, the JS/TS SDK, direct OTLP — never pass through this SDK's app-root marking, so such a trace is reachable in practice, not theoretical.scope='traces'is one-observation-per-trace by definition, so evaluating an arbitrary descendant instead is not obviously better — but the drop is silent, which is why it is stated here.A
scope='traces'filter that is not a JSON array cannot be merged with the root condition, sorun_asyncnow rejects it before the fetch loop. Validating inside the loop would be swallowed by itsexcept Exceptioninto acompleted=Falseresult whose resume token carries an empty timestamp bound — a guaranteed 400 on the next run, with nothing pointing at the filter.scope='observations'forwards the caller's filter unchanged and is unaffected.Behavioural change for
scope='traces'Items passed to the mapper go from
TraceWithFullDetails— a whole trace, carrying theobservations,scoresandmetricsarrays — to oneObservationV2per trace, and only ever the root observation. Mappers that consumed the fullobservations/scoreslists on a trace cannot be adapted field-for-field: those collections are gone rather than renamed. Two field-name traps follow from the item now being an observation:item.idis the observation ID. The trace ID isitem.trace_id, and that is also what score-create calls attach to (see_get_item_id).item.user_idanditem.tagsdo still work — the v2 observation carries them denormalized from the parent trace.Mappers that read the root observation's
input,output,metadata,model,usagework as-is. This change is the same direction the issue reporter suggested (workaround: "fetch v2 observations grouped by traceId, root observation for trace-level io") and aligns withlangfuse/langfuse-python#1867(batch evaluation marked as legacy, v2-only path).scope='observations'items now also come from/api/public/v2/observationsinstead of the v1 read endpoint; items areObservationV2instances withinput,output,metadata,model,usageandtrace_contextpopulated.Known limitations (disclosed, not fixed here)
metadatavalues to 200 characters by default, withexpandMetadatareturning full values. Measured on the self-hosted 4.42.0 events_only stack, a 300-character value came back untruncated both with and withoutexpandMetadata, so the documented truncation was not observed there; behaviour may differ across server versions. The runner does not passexpandMetadataeither way. If a deployment does truncate, passing afetch_expand_metadata-style option throughrun_batched_evaluationwould be the follow-up.GET /api/public/scoresandGET /api/public/v2/scoresmeasured 404 on 4.42.0. Scores created by runs in this PR were verified by reading the events store (ClickHouse) directly; on Langfuse Cloud the UI is the read surface. (An earlier revision of this description said scores were visible atGET /api/public/v2/scores; that endpoint is not readable on this deployment today.)scope='traces'resume boundary: resume filters on the representative observation'sstart_timewithstartTime > T. Narrowing to roots means that representative is now always the root, so the bound is the trace's own start rather than an arbitrary descendant's — a trace whose root started at or beforeTno longer re-enters a resumed run through a later child. A trace whose root started afterTis of course still picked up, which is the intended behaviour. The residual gap is thatTis the last processed root's start, so a trace that overlapsTcan be evaluated twice across two runs; the v2 filter grammar has no not-equal operator ontraceIdand no trace-level read, so there is no clean fix at the API layer.isRootObservationagainst a live server: verified on a self-hosted v4events_onlydeployment at621ccc83— see the root-selection section below.Type of change
scope='traces'; see above)Verification
Commands run on
621ccc83against65392c73(base_sha, current upstreammain):The six cases covering the root narrowing and the filter rejection were also run
against the previous implementation of
_translate_trace_filter(the page-localbehaviour, without the
run_asyncvalidation): all six fail there and pass on131d296c/621ccc83, so they pin this change rather than restating it.End-to-end against a real Langfuse v4.42.0 events_only docker-compose stack (postgres + clickhouse + redis + minio + langfuse-web + langfuse-worker):
65392c73):batch_evaluationreturns 0 scores; every retry fails with HTTP 404 events_only onGET /api/public/v3/tracesand the v3 observations endpoint.045f98aa): seeded 3 traces via SDK; ranbatch_evaluationwithscope='observations';total_items_processed=22,total_scores_created=15,completed=true. New scores visible atGET /api/public/v2/scores.Full end-to-end log:
oss-agent/logs/20260923T103452-nightly-5f51a956/e2e-after-fix.log.Root selection, measured on the same stack (
621ccc83)The deployment was confirmed events_only before any conclusion was drawn from
it:
GET /api/public/tracesandGET /api/public/observationsreturn 404events_only, whileGET /api/public/v2/observationsreturns 200.A trace with one root and twelve children, read back at
page_size=2so theroot cannot share a page with all of its children:
The pre-fix path tracked the server's ordering — across repeated runs the root
was sometimes on page 1, when pre-fix also picked it, and sometimes not, when
pre-fix picked a child instead. The root-filtered path returned the root exactly
once in every run. That variance is why the e2e test records the observed
ordering instead of asserting a fixed one.
The filter was also checked in both directions, so it is not a no-op:
isRootObservation = truereturns only the flagged root, andisRootObservation <> truereturns the twelve children and excludes it.New file
tests/e2e/test_batch_evaluation_root_selection.py(5 tests, passing)covers all of the above: the filter being honoured in both directions, the root
winning across cursor pages,
run_batched_evaluation(scope='traces')handing themapper the root's payload rather than a child's, and the two cases that break the
"one root per trace" assumption —
isRootObservation=trueThe no-root case is written through the OTLP endpoint because the observation
write endpoints refuse creates on an events_only deployment:
POST /api/public/observationsanswers 405, andPOST /api/public/ingestionanswers
Event type "SPAN" is not accepted ... only accepts score events. Thaterror names the OTLP path as the events_only-compatible route, which is also the
realistic route for a trace ingested by another client.
The e2e file needs no shard registration —
scripts/select_e2e_shard.pyassignsit to shard 0.
Remaining paths, re-verified on the same stack (
621ccc83)Everything else this PR touches was re-run end to end on the same deployment — a
4-trace corpus (root as a GENERATION carrying
model, 2 child spans each, a300-character metadata value on every root):
scope='observations'e2einput/output/metadatapopulated,modelon every GENERATION, 12 scores createdobservation_idset on all 12,trace_idsetscope='traces'e2eobservation_idempty on all 4, one per trace — attached to the trace, not the root observationnamename=selected exactly the one named trace (viatraceName)idid=selected exactly that trace (viatraceId)timestamptimestamp > Tselected exactly the corpus (viastartTime)namefilter stayed an observation-name filter: 1 item, the matching childmax_itemsmid-streamfetch_batch_size=1,max_items=1, 4 traces: processed 1,has_more_items=TruestartTime > Tat the 2nd-earliest root: resumed run evaluated exactly the 2 later traces, no duplicates,completed=True_additional_trace_tagsTwo recorded observations that correct earlier statements in this description:
max_itemsstop reportscompleted=Truewithhas_more_items=Trueandno resume token; only the fetch-failure path produces a token. A user
wanting to resume a limit stop constructs the token from the result shape
themselves. That is pre-existing semantics, unchanged by this PR, and worth
knowing when reading
has_more_items.Scores were verified by querying the events store (ClickHouse) directly because
both public score read endpoints are gated on events_only deployments (see
limitations). Verification log with all thirteen checks: run
20261010T112000-morning-c4d5e6f7in the private work repository.Checklist
code_review.md.tests/unit/test_batch_evaluation_fetch.py, 6 of them covering the root narrowing and the filter rejection, plus 5 e2e tests intests/e2e/test_batch_evaluation_root_selection.pyrun against a live events_only deployment)..env.templateif needed — therun_asyncdocstring and the module-level filter constants now describe the root narrowing; no user-facing config touched.langfuse/api/was not touched.ObservationV2is imported from the generatedlangfuse.apinamespace.